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(57) ABSTRACT 

A method and apparatus in which the appearance and 
functionality of a virtual remote control panel on a web page 
is controlled by an applet downloaded in connection with the 
web page. Because the appearance and functionality of the 
virtual control panel is controlled by the applet, it is easy to 
change. The invention also includes a set of methods or 
functions (an API) that can be used by the applet to interface 
to the video source. This API uses a PIP abstract Java class 
and a PIPInfo abstract Java class, where "PIF' stands for 
"picture in picture." 
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Example of a Web page with virtual 
controller and displayed video content 



02/02/2004, EAST Version: 1.4.1 



Patent Application Publication Mar. 20, 2003 Sheet 1 of 23 



US 2003/0056215 Al 




Patent Application Publication Mar. 20, 2003 Sheet 2 of 23 US 2003/0056215 Al 




02/02/2004, EAST Version: 1.4.1 



Patent Application Publication Mar. 20, 2003 Sheet 3 of 23 US 2003/0056215 Al 




02/02/2004, EAST Version: 1.4.1 




02/02/2004, EAST Version: 1.4.1 



Patent Application Publication Mar. 20, 2003 Sheet 5 of 23 US 2003/0056215 Al 




02/02/2004, EAST Version: 1.4.1 




02/02/2004, EAST Version; 1.4.1 




02/02/2004, EAST Version: 1.4.1 



Patent Application Publication Mar. 20, 2003 Sheet 8 of 23 



US 2003/0056215 Al 




Patent Application Publication Mar. 20, 2003 Sheet 9 of 23 



US 2003/0056215 Al 




02/02/2004, EAST Version: 1.4.1 



Patent Application Publication Mar. 20, 2003 Sheet 10 of 23 US 2003/0056215 Al 




02/02/2004, EAST Version: 1.4.1 



Patent Application Publication Mar. 20, 2003 Sheet 11 of 23 US 2003/0056215 Al 



r 



CD CO J~ ^ ^. 



c 
o 

*J « 
ro <t> 
so 

| J 

Is 

K g 

u 

S£ 
s § 

CO I- 

8> 

O 



CO 

<D - 



~ 0) 



0) 
0) 

° «^ 

CO c 

E ^ O 



8 



§1 
o | 

£ J C 

<o o p 
* S 5 
3 8 & 

a) ^ JE 
co a. 

Q_ 



1^ 

T3 CO 
« $ 

0) Q- 

> .* 



8 



a: 



'> 

Q. 
(0 

b 



O -° tO 

to ? o 

© £ )- 

to ° jS 
>, a) c 

5 Sj? | 
5 ° g- ■ 

o *~ to 

^ T3 O ^ 

is £ o 
0) c ^ 



c 8 .£ 

q. a. o 
O 




5 jg 



5 5 '8 S 



8 



S, m 2 

"XS g.g 

^-2 8 ^ if 

* = si ss 
« § I * ■ 



7 



■o $ £ "EL ~ 

™ H- C * O 



°§ 8S S£ 8 

P re «> 2 o.>p 
| g.1 A 8. 



CD 



CD 

ll 



02/02/2004, EAST Version: 1.4.1 



Patent Application Publication Mar. 20, 2003 Sheet 12 of 23 US 2003/0056215 Al 




J2 re 

CL-p -O CD 

C £ u- o 

:r a> O 



> r? 
I- 



CD 0) 
O) = 

c o 

x: c 

9 8 




to 
E 

— Si. o- 

O §|?5 

lis* 

-a E ^ 2 

c ;r CD £ 



CO o 

-e a) 

— CO 

I- 

=5 ° 

<d a> 



§28 



05^ 



0) 

c 

c ^_ 

(0 o 



CO 

c 
c 

CO 



a. o 



c _ 

? Si 

Ji! - 

a> o 

■s 0) 
to ~ " 
c „ a> 



CD *o 
O o 

o 



- o 



e g cu 



4= c 



0) o - 



15 



/it n ™ 



CD Q- CD a. 



CO 

> Sf 

CD <d 
0) = 

c o 

9 8f 



02/02/2004, EAST Version: 1.4.1 



Patent Application Publication Mar. 20, 2003 Sheet 13 of 23 US 2003/0056215 Al 




0^ 

Ll 





Li. 



02/02/2004, EAST Version: 1.4.1 



Patent Application Publication Mar. 20, 2003 Sheet 14 of 23 US 2003/0056215 Al 




O 
0) 

O 



00 



CD 

c 

Q. 

E 

CO 
X 

LU 



02/02/2004, EAST Version: 1.4.1 



Patent Application Publication Mar. 20, 2003 Sheet 15 of 23 US 2003/0056215 Al 



o 



O 
CD 

CO 

> 

CO 

CO 
03 

sz 

2 

(U 
Q- 

Q- 
C 



1 

CO 

o 

cu 
■g 

"> 

CO 

o 
o 
O 



CO 
0) 

to 
c 

"2 
o 
o 
O 

? C ^ 

o 

» 8% 

^> -S> 

co co £ 
.S>-o 5 
^- o 8 

CD > ^ 
CD O O 
CO := 
* — .Q -Q 
CO 3 3 

-a a. a 

o 

sz 

"8 



CO 
(D 

° 8 

o S 
-o o 

% I 

£ 2 

a> 
> 

ca 



O 

* 

o 

■2. 

el 

00 ® 
■ X5 

u_ ^ 

Q. 

E 

CO 
X 

LLJ 



02/02/2004, EAST Version: 1.4.1 



Patent Application Publication Mar. 20, 2003 Sheet 16 of 23 US 2003/0056215 Al 




0) 



8 



OT (0 <S 

c c .S 
.2 -5 P 

ego 
a O O 

<0 0) <*> 

0) O 

° <D 0 

> > 



0) 0) 

I = 

5 to 

O CO 



(D <D 

O CO 

0) CD 

> > 



ft 
If 

CO O 

> > 



(J 

OO 



C (0 CO 

c c c 



CO CO 

c c 



(0 (0 CO CO 

e c c c 



CO 

> 

"to 
z 



o 

CD 

J? 
O 

O 
(D 

■ E 

il e 
a. 

Q) 
Q. 

E 

CO 
X 
LU 



02/02/2004, EAST Version: 1.4.1 



Patent Application Publication Mar. 20, 2003 Sheet 17 of 23 US 2003/0056215 Al 




C 

1_ 

0) 

c 

3 
I- 

i ° 

CO 

o 

s 



0 
c 
c 

CO 

"(D 

(Ji .c 



O 
> 

o 

Q_ 




c 

O 

o 

-4— • 



c 

O 



(0 



0) 

c 

CD 
CO 

4— • 

c 

CO 
.*-* 
CO 

c 



2.E 

1 

CO 

c 

To 
O 



aj co 
to .E 

■o 0 
o ao 

CO o 
CO <u 



8=5 



</> 

CD 
■♦— > 

05 

C 

'"5 
o 
o 
O 

<0 
, »+— 

a) 
Q 

O) 
0 
O) 

c 

CO 

o 
Zi 

a. 



LL 



02/02/2004, EAST Version: 1.4.1 



Patent Application Publication Mar. 20, 2003 Sheet 18 of 23 US 2003/0056215 Al 




a 

c 
o 

<D 

^ "O 

3> 
a) q5 

• (0 

El 

> 
o 

JQ 
3 
Q. 




O 

3 
Q. 











& 




c 












O" 








I 




0) 












H 








c 


CD 


Oi 




Li. 


cr 

CD 




LL 




-*-» 
CD 




(0 




TJ 




O 




> 
















Q. 



02/02/2004, EAST Version: 1.4.1 



Patent Application Publication Mar. 20, 2003 Sheet 19 of 23 US 2003/0056215 Al 




02/02/2004, EAST Version: 1.4.1 



Patent Application Publication Mar. 20, 2003 Sheet 20 of 23 US 2003/0056215 Al 



CO 
CO 
(0 

73 
13 

E _ 
o o 

"1 

« s 

"S « 

^ *— 
OJ O 

'c ts 

CO Q) 
(O .Q 

,c o 

$ ® 

"E C3 
O i_ 
O CD 

o _c 

CO CD 

2 r 



o 

o 

O 
• 

cd 

CO 

O 



CL 

.52 

T5 
O 
CD 

-a 
> 
c 

CO 

B 

CO 

c 



o 

8 

>«-* 

CD 
(0 



CD 
TJ 
O 
O 

is 

c 

la 
O 



CD 
» 

CO 

c 
•a 

o 
o 
o 

0 
Ui 

c 

CO 

t> 



•o 

2 «> 

o> "co 

O 
O 

a 

(D 
</> 

■g 

"o 
> 

o 

Q. 



>* 
£ 
a. 

CO 
O 

> 

CO 
0} 

"56 (0 

.s s 

P « 
O .S 

o p 
° o 

IS 



CD 

<d a: 

a) 
2 

CD 

TO 



8 

CD 
# > 

oa 
c 

15 
O 



0) 
CO 

c 

iE 

o 
o 

.£? CD 

(0 
•*-> 

o 

(D 

o 

Q. 



02/02/2004, EAST Version: 1.4.1 



Patent Application Publication Mar. 20, 2003 Sheet 21 of 23 US 2003/0056215 Al 




Q. 
Q- 

t- CL C 

$1%. 

o 
> 

o 

3 
CL 



02/02/2004, EAST Version: 1.4.1 



Patent Application Publication Mar. 20, 2003 Sheet 22 of 23 US 2003/0056215 Al 



o 

o 

c 
o 

c 

CO 

o 
o 
.a 

Q c 
a) 2 

(D CD 
9> O 

c CO 

■o c 

l.£ 

cd a> 

> > 

CO CO 

c c 

<D CD 

«♦-• 
CO CO 

> > 

O- CL 



13 o 
£ £ 
<D SL 

c c 
co <S 

S 0) 

"?5 5 
c c 

8 § 

oy 

a> 

W CD 

CD CD 

> > 

c c 
"5 "5 

> > 

CD 0) 

> > 

CO co 
c c 

SI 

CO CO 
<D 0 

"co 15 

> > 

a. a. 



<D 

c 
c 

(0 



o 
c 
<D 

D" 
CD 



"CD "CD 

I = 

CO CO 

«§ 

CD 0) 

O w 

CD CD 
> > 

CO CO 

c c 



c fD 
cr © 



y= a) 



c c c 



CO 

c 

■g 
*o 
> 



CD CD CD CD 

> > > > 

'^3 '-^ .»_> -»— • 

CO CO CO CO 

c c c c 

o o o o 

B B B B 

(I) (A U) (O 

CD CD CD CD 

CO CO CO CO 

> > > > 

a a a a 



CD 
C 

t 

O 
o 

CD 

"> 



a. 

= 1 



U_ m "55 CD 



ts 

D) 

c 

CO 

o 

<D 
CO 

1 8 S 



a3 a> 
CO 0 

CD CD 

> > 

TO CO 
C C 

5 O 

> > 

CD CD 

> > 

CO CO 

c c 
o o 

iS 2 

CO CO 
CD CD 
CO CO 

> > 
a. a. 



o 
o 

o - w 

CD £ JO 

a o a) 
oQQ 

cd o O 

> CD CD 



c 



CO CO 

c c 

> > 1 

4— ' 

CO CO CO 

c c c 

g o o 

+j '*i 

B B B 

(0 CO CO 

0) (D CD 
+-* 

CO CO CO 

> > > 
o. a a. 



Q. 

< 

o 

il I 
z 



02/02/2004, EAST version: 1.4.1 



Patent Application Publication Mar. 20, 2003 Sheet 23 of 23 US 2003/0056215 Al 



CO 



? JL if 

<2 o "o 

3 O O 

t roo a 



CD jl> 

O) Q. 

CO Q_ 

a. (o 

-g te 

5 2 

N u O 

T" O O 



LL 



^ O Q- 

\7 V V V 



CO 



CO 
Z3 

(u -ti 
LU O 



o 

CM 



02/02/2004, EAST Version: 1.4.1 



US 2003/0056215 Al 



1 



Mar. 20, 2003 



TV PIP USING JAVA API CLASSES AND JAVA 
IMPLEMENTATION CLASSES 

RELATED APPLICATIONS 

[0001] This application is related to the following co- 
pending U.S. patent applications, which are filed concur- 
rently with this application and which are herein incorpo- 
rated by reference. 

[0002] 1. U.S. patent application Ser. No. , 

entitled "Applet and Application Display in Embed- 
ded Systems Using Child and Orphan Graphics 
Contexts" of Rajesh Kanungo; 

[0003] 2. U.S. patent application Ser. No. , 

entitled "Display Widget Interaction in Embedded 
Systems Using Child Graphics Contexts" of Rajesh 
Kanungo; 

[0004] 3. U.S. patent application Ser. No. _ 
entitled "Method and Apparatus For Modal Dialog 
Box Management In Applets On Information Appli- 
ances" of Rajesh Kanungo and Juan Carlos Soto, Jr.; 

[0005] 4. U.S. patent application Ser. No. , 

entitled "TV PIP Applet Using PIP Framework and 
Implementation" of Rajesh Kanungo; and 

FIELD OF THE INVENTION 

[0006] This application relates to a method and apparatus 
for a user interface control and, specifically, to a method and 
apparatus that allows an applet to control the appearance and 
functionality of a user interface on a web page. 

BACKGROUND OF THE INVENTION 

[0007] Television sets and VCR players are traditionally 
controlled by a physical device, such as a handheld remote 
unit or directly by physical controls mounted on the set or 
player itself. It is often difficult for a designer of a physical 
remote unit to correctly anticipate and design the best 
possible user interface for his unit. Furthermore, the "best" 
user interface for a remote may vary between various users 
and may vary, depending on the type of activities that a 
particular user desires to perform. Unfortunately, physical 
control units lack a flexibility of user interface design. 

[0008] In addition, physical remote units tend to get mis- 
placed or lost. See, for example, U.S. Pal. No. 5,294,915 to 
Owen, issued Jun. 30, 1993, relating to locating a lost 
physical remote control unit. It would be desirable never to 
have to worry about lost physical remotes. 

[0009] This problem is compounded in certain types of 
consumer electronics devices that do not contain provisions 
for reprogramming the video unit, so that it is not possible 
to easily change the programming of the video unit. What is 
needed is a flexible type of user interface that allows the 
designer to change the functionality and the look and feel of 
a remote control. 

SUMMARY OF THE INVENTION 

[0010] A described embodiment of the present invention 
allows the user to control the display of video data (sent 
from a television set or other video source) via a "virtual 
remote control" displayed on a World Wide web page being 
viewed by the user. A described embodiment of the present 



invention is implemented in a set top box environment, 
although the present invention could also be implemented in 
a browser capable of running Java applets and executing on 
any appropriate computer or data processing system. 

[0011] In the described embodiment, the user downloads 
and views a web page containing a Java applet. When 
executed, the Java applet displays a "virtual remote control" 
panel. Because the look and feel of the virtual remote control 
panel is entirely defined in the downloaded applet, it is easy 
to change the configuration and appearance of the virtual 
remote control panel. No programming changes are required 
in the set top box (or the video source) to implement changes 
in the user interface of the remote. 

[0012] The described embodiment of the present invention 
includes a PIP (Picture in Picture) API (Application Pro- 
gramming Interface). This API contains a number of pre- 
defined Java methods that can be called by the downloaded 
applet to control the display of the video data. Thus, the 
functionality of the virtual remote can also be easily 
changed, since the virtual remote is implemented by the 
applet making calls to the PIP API. Thus, the applet can 
display and operate a virtual remote control having any 
functionality consistent with the functionality allowed by the 
PIP API. The term PIP refers to the fact that the video data 
is displayed as a ''picture within a picture" on the web page. 
It does not necessarily refer to the concept of displaying one 
video picture inside another video picture, although such a 
conventional video picture within a video picture could also 
be implemented in accordance with the invention. 

[0013] In accordance with the purpose of the invention, as 
embodied and broadly described herein, the invention 
relates to a method of controlling display of video data, 
comprising: creating, by an applet, a PIP object and a 
PIPInfo object; sending the PIPInfo object to the PIP object 
to initially control display of the video data within a web 
page; receiving, by the applet, user input via a virtual control 
panel displayed by the applet on the web page; and calling, 
by the applet, virtual control API functions in accordance 
with the user input to communicate with the PIP object and 
the PIPInfo object to control the display of video data within 
the web page. 

[0014] A fuller understanding of the invention will 
become apparent and appreciated by referring to the follow- 
ing description and claims taken in conjunction with the 
accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0015] The accompanying drawings, which are incorpo- 
rated in and constitute a part of this specification, illustrate 
several embodiments of the invention and, together with the 
description, serve to explain the principles of the invention. 

[0016] FIG. 1(a) is a block diagram of a data processing 
system in accordance with one embodiment of the present 
invention. 

[0017] FIG. 1(b) is a block diagram of a data processing 
system in accordance with one embodiment of the present 
invention. 

[0018] FIG. 2 shows an example of a set top box system, 
including a remote unit, and a video display. 
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[0019] FIG. 3(a) shows an example of a display device, 
including a video area and a virtual remote control panel 

[0020] FIG. 3(b) shows an example of a display device, 
including a small video area, additional content, and a 
virtual remote control. 

[0021] FIG. 3(c) shows an example of a display device, 
including two video areas and a virtual remote control panel. 

[0022] FIG. 3(d) shows an example of a display device, 
including two video areas, one inside the other, and a virtual 
remote control panel. 

[0023] FIGS. 4(o)-4(0 show examples of pull-down 
menus for select buttons on the virtual remote control of 
FIG. 3. 

[0024] FIG. 5(a) shows an example of Java classes used 
in the described embodiment of the present invention. 

[0025] FIG. 5(b) shows a block diagram of the described 
embodiment of the present invention, including an applet, 
instances of the classes of FIG. 5(a), and native code 
controlling a display device. 

[0026] FIG. 6 is a flow chart of a method performed by the 
applet of FIG. 5(b). 

[0027] FIGS. l(a)-l(e) are exemplary flow charts showing 
actions taken by the applet in response to various user 
actions in the described embodiment of the present inven- 
tion. 

[0028] FIGS. 8(a)-8(c) are examples of, respectively, 
objects of a PIP class, a Tunerlnfo class, and a Videolnfo 
class. 

[0029] FIGS. 9(a)'9(h) are flow charts of exemplary meth- 
ods in the Tunerlnfo class and the Videolnfo class. 

[0030] FIGS. 10(a)-10(0 are flow charts of exemplary of 
methods in the PIP class that are called by the virtual remote 
control applet. 

[0031] FIG. 11 is an exemplary list of native code func- 
tions called by the objects of FIG. 8. 

[0032] FIG. 12 is an example of HTML for a Web page 
that contains an example of the virtual remote control applet. 

DETAILED DESCRIPTION OF EMBODIMENTS 
[0033] I. General Discussion 

[0034] Many conventional World Wide Web browsers 
include a Java virtual machine and are capable of executing 
Java applets. A Java applet is a computer program written in 
the Java programming language that is designed to execute 
"within" a World Wide Web browser. Such applets are often 
started when a user clicking on a button or link in a displayed 
Web page. Once started, an applet executes inside the 
browser, performing whatever tasks it has been programmed 
to perform. "Java" is a trademark of Sun Microsystems, Inc. 
in the United States and other countries. 

[0035] FIG. 1(a) is a block diagram of a data processing 
system 100 in accordance with one embodiment of the 
present invention. Data processing system 100 can be, for 
example, a set top box 101 connected to a display device 
130, such as, for example, a television set or some other 
appropriate display device. Data processing system 100 can 



also be, for example, any data processing system that does 
not allow the user to have access or control of its operating 
system or that does not allow the user to warm boot the 
system at certain times. 

[0036] System 100 includes a processor 102 and a data 
storage area (e.g., a memory) 104. Data storage area 104 
includes certain well-known types of information, such as 
operating systems 112 and data and data structures 114. In 
the embodiment shown, data and data structures 114 include, 
for example, web pages capable of being displayed by a 
browser 106. Data and data structures 114 can also include, 
for example, applet software capable of being executed by 
a Java virtual machine (JVM) 108. Storage area 104 pref- 
erably also includes software (not shown) for communicat- 
ing with a network connection 103, which can be a LAN, 
WAN, intranet, or the internet. 

[0037] A person of ordinary skill in the art will understand 
that system 100 may also contain additional elements, such 
as input/output lines; additional or second input devices, 
such as a keyboard, a mouse, and a voice input device; and 
display devices, such as a display terminal. System 100 may 
also include a computer readable input device (not shown), 
such as a floppy disk drive, CD ROM reader, a chip 
connector, a chip connector, or a DVD reader, that reads 
computer instructions stored on a computer readable 
medium, such as a floppy disk, a CD ROM, a memory chip, 
or a DVD disk. System 100 also may include application 
programs, operating systems, data, etc., which are not shown 
in the figure for the sake of clarity. 

[0038] In the following discussion, it will be understood 
that the steps of methods and flow charts herein discussed 
herein can be performed by processor 102 (or another 
appropriate processor) executing instructions stored in stor- 
age area 104 (or other appropriate memories or storage 
areas). It will also be understood that the invention is not 
limited to any particular implementation or programming 
technique and that the invention may be implemented using 
any appropriate techniques for implementing the function- 
ality described herein. The invention is not limited to any 
particular programming language or operating system. 

[0039] The instructions in storage area 104 may be read 
into storage area 104 from a computer-readable medium (not 
shown). Execution of sequences of instructions contained in 
main memory causes processor 102 (or a similar processor) 
to perform the processes described herein. In alternative 
embodiments, hard -wired circuitry may be used in place of 
or in combination with software instructions to implement 
the invention. Thus, embodiments of the invention are not 
limited to any specific combination of hardware circuitry 
and software. 

[0040] FIG. 1(a) shows a virtual machine (such as a Java 
virtual machine) 108 executing on system 100. FIG. 1(b) is 
a block diagram of a virtual machine that is supported by 
system 100 of FIG. 1(a), and is suitable for implementing 
the present invention. When a computer program, e.g., a 
computer program written in the Java programming lan- 
guage, is executed, source code 160 is provided to a com- 
piler 170 within compiletime environment 155. Compiler 
170 translates source code 160 into bytecodes 180. In 
general, source code 160 is translated into bytecodes 180 at 
the time source code 160 is created by a software developer. 

[0041] Bytecodes 180 may generally be reproduced, 
downloaded, or otherwise distributed through a network, 
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e.g., network 103 of FIG. 1(a), or stored on a storage device 
such as primary storage 104 of FIG. 1(a). Io the described 
embodiment, bytecodes 180 are platform independent. That 
is, bytecodes 180 may be executed on substantially any 
computer system that is running on a suitable virtual 
machine. 

[0042] Bytecodes 180 are provided to a runtime environ- 
ment 108, which includes virtual machine 190. Runtime 
environment 108 may generally be executed using a pro- 
cessor or processors such as processor 102 of FIG. 1(a). 
Virtual machine 109 includes a compiler 192, an interpreter 
194, and a runtime system 196. Bytecodes 180 may be 
provided either to compiler 192 or to interpreter 194. 

[0043] When bytecodes 180 are provided to compiler 192, 
methods contained in bytecodes 180 are compiled into 
machine instructions. In one embodiment, compiler 192 is a 
just- in-time compiler, which delays the compilation of meth- 
ods contained in bytecodes 180 until the methods are about 
to be executed. When bytecodes 180 are provided to inter- 
preter 194, bytecodes 180 are read into interpreter 194 one 
bytecode at a time. Interpreter 194 then performs the opera- 
tion defined by each bytecode as each bytecode is read into 
interpreter 194. That is, interpreter 194"interprets" byte- 
codes 180, as will be appreciated by those skilled in the art. 

[0044] When a method is invoked by another method, or 
is invoked from runtime environment 108, if the method is 
interpreted, runtime system 196 may obtain the method from 
runtime environment 108 in the form of sequences of 
bytecodes 180, which may be directly executed by inter- 
preter 194. If, on the other hand, the method that is invoked 
is a compiled method that has not been compiled, runtime 
system 108 also obtains the method from runtime environ- 
ment 108 in the form of a sequence of bytecodes 108, which 
then may go to activate compiler 192. Compiler 194 then 
generates machine instructions from bytecodes 180, and the 
resulting machine-language instructions may be executed 
directly by processor 102 (or other appropriate processors). 
In general, the machine-language instructions are discarded 
when virtual machine 109 terminates. The operation of 
virtual machines or, more particularly, Java virtual 
machines, is described in more detail in The Java Virtual 
Machine Specification by Time Lindholm and Frank Yellinx 
(ISBN 0-201-63452-X), which is incorporated herein by 
reference. 

[0045] II. Detailed Discussion 

[0046] FIG. 2 shows an example of a set top box system, 
including a set top box 101 connected to a television 130 and 
a remote unit 120. Remote unit 120, which is used as an 
input device, is not shown to scale. An actual remote unit is 
generally of a size appropriate to fit in a user's hand. 
Similarly, set top box 101 and television 120 are not 
necessarily shown to scale. It will be understood that It will 
be understood that the button arrangement shown on remote 
unit 120 are for purposes of example only and are not to be 
interpreted in a limiting sense. Remote unit 120 includes a 
"web" button, a "mail button, a "home" button, a "goto" 
button, a "fave" button, a "back" button, a "ctT button, 
arrow buttons, and a "select" button. As will be appreciated, 
a conventional use of back button is to display a previously 
displayed web page. Similarly, a conventional use of "goto" 
button is to allow the user to specify and view a new web 
page. The arrow buttons allow the user to move a cursor 
between various elements on web page 304. 



[0047] In FIG. 2, a video area 304 and a virtual remote 
control panel 310 are displayed on a web page displayed on 
display device 130. It will be understood that both of 
elements 304 and 310 arc generated on a web page under 
control of an applet (not shown), such as applet 110 of FIG. 

1M- 

[0048] Network connection 103 allows the set top box to 
communicate with a network, such as the internet, an 
intranet, a LAN or a WAN. Network connection 103 can also 
be a wireless link. Certain embodiments of the present 
invention can also be used with a keyboard or with an 
infrared or other wireless keyboard (not shown). 

[0049] Video line 207 provides a source of video input to 
set top box 101. Line 206 couples the set top box 101 to the 
display device 130 to convey video and audio output signals 
to the display device 130. The signals on line 206 could be, 
for example, one or more of, S-video signals, composite 
signals, component signals, or audio signals, HDTV signals. 

[0050] FIG. 3(a) shows an example of a display device 
130 on which a web page 302 is displayed. Web page 302 
is displayed in accordance with HTML or a similar descrip- 
tive language or convention (see example of FIG. 12). Web 
page 302 includes a video area 304 and a virtual remote 
control panel 310. In the described embodiment, certain 
elements of the web page, such as area 304 and virtual 
control panel 310 are implemented as widgets generated by 
the applet, but they could also be implemented as a part of 
the page described by HTML. In the example, video data is 
being displayed in video area 304 and user interface ele- 
ments are being displayed in virtual control panel 310. 

[0051] The user interface elements in this exemplary vir- 
tual control panel include an ON/OFF selector button 312 
(here, ON is selected); a TV/Video selector button 314 (here, 
TV is selected); a Broadcast/cable selector button 316 (here, 
broadcast is selected), a size button (here, 100% is selected), 
and a channel indicator 320 (here, the set is tuned to channel 
002). It will be understood that the user interface elements 
shown herein an provided for purpose of example only and 
are not to be interpreted in a limiting sense. Any appropriate 
control elements, including but not limited to those shown 
herein, can be used in connection with the preseDt invention. 

[0052] FIG. 3(b) shows an example of a display device, 
including a small video area 304', additional content (for 
example, an advertisement 320 and other text 322) and a 
virtual remote control panel 310. As shown, the user has set 
the video display to be a size of 50%. In the embodiment of 
FIG. 3(6), the applet displays other content in the remaining 
space on the page. In yet another embodiment, the applet 
may always choose to display the video in a certain size area 
304 and always display advertisements or other appropriate 
content. 

[0053] FIG. 3(c) shows an example of a display device, 
including two video areas 304" and a virtual remote control 
310'. In the example, the virtual control panel includes two 
channel indicators, one for each video area 304". 

[0054] FIG. 3(d) shows an example of a display device, 
including two video areas, where video area 305 is inside 
video area 304'", and a virtual remote control panel. In the 
example, the virtual control panel includes two channel 
indicators, one for each video area 304"' and 305. 
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[0055] FIGS. 4(a)-4(d) show examples of pull-down 
menus for select buttons on the virtual remote control of 
FIG. 3. As shown, the user selects a button to view a list of 
possible choices and then selects one of the choices. FIG. 7 
discusses actions performed by the applet in response to user 
input It should be understood that other known types of user 
interfaces can be used to implement that video functions 
shown herein (or similar video functions). 

[0056] FIG. 5(c) shows an example of Java classes used 
in the described embodiment of the present invention. It 
should be understood that the classes presented herein are 
presented by way of example and are not to be taken in a 
limiting sense. The invention can also be implemented using 
similar object oriented programming languages and is not 
necessarily limited to Java. The exemplary classes include 
two abstract Java classes: an abstract PIP class and an 
abstract PIPlnfo class. A PIPImple mentation class is a sub 
class of the abstract PIP class. A Tunerlnfo class and a 
Videolnfo class are sub classes of the PIPlnfo class. As will 
be described below, instances of the PIPImplementation 
class use instances of the PIPinfo class (I.e., Tunerlnfo and 
Videolnfo objects). 

[0057] FIG. 5(b) shows a block diagram of the described 
embodiment of the present invention, including an applet 
110, instances 502, 504, 506 of the classes of FIG. 5(a), and 
native code 508 controlling a display device 130 via a video 
chip 510 In the described embodiment, the video chip is an 
IGS Cyberpro 5000 chip manufactured by IGS Technolo- 
gies, Inc. Video chip 130 is connected to and controls a 
display device 130. Native code 508 is written in, for 
example, the C++ programming language in a manner 
appropriate to control the video chip. The exact nature and 
specifications of native code 508 will depend on the specific 
video chip employed. 

[0058] FIG. 6 is a flow chart of a method performed by the 
applet of FIG. 5(b). Steps 602 and 603 are actually steps 
performed by a browser to download a web page containing 
the applet and begin execution of the applet. In step 606, the 
applet creates an instance of each of the PIPImplementation 
class, the Tunerlnfo class, and the Videolnfo class. When 
each instance is created, it will initialize itself with prede- 
termined default values. In step 610, the applet registers to 
receive key presses from an operating system. In step 612, 
the applet displays virtual control panel 310. If the set top 
box is receiving keyboard input (for example, from an 
infrared keyboard, or other appropriate keyboard), in step 
314, the applet sets a trap for numeric key strokes in a key 
stroke event handler (which indicate a channel change). In 
step 616 the applet sets a trap for other key presses in the key 
stroke handler, which are used to change other video 
attributes of the controller. Step 616 is performed, for 
example, in embodiments having a keyboard with numeric 
keys. The embodiment shown in FIG. 2 uses the arrow keys 
on the remote to choose between numeric channel values 
displayed by the applet on the virtual remote panel. 

[0059] If the applet wants to start in TV mode in step 620, 
it sends the Tunerlnfo object to the PIPImplementation 
object in step 620. Otherwise (not shown), it sends the 
instance Videolnfo to the PIPImplementation object. 

[0060] FIGS. 7(a)- 7(e) are exemplary flow charts showing 
actions taken by the applet in response to various user 
actions in the described embodiment of the present inven- 



tion. FIG. 7(a) shows steps when the user changes the video 
source from TV to Video (e.g., from TV to a VCR). FIG. 
1(b) shows steps when the user changes the display size. 
FIG. 7(c) shows steps when the user switches the video 
display within the web page ON or OFF. FIG. 1(d) shows 
steps when the user changes the frequency type (e.g., 
broadcast or cable). FIG. 7(e) shows steps when the user 
changes the channel. It will be understood that the methods 
of FIG. 7 are exemplary only and are not to be interpreted 
in a limiting sense. 

[0061] FIGS. 8(a)-8(c) are examples of, respectively, 
objects of a PIPImplementation class, a Tunerlnfo class, and 
a Videolnfo class. As shown in FIG. 8(a), a Tunerlnfo object 
contains data identifying a current PIP type (i.e., "Tuner"), 
coordinates of the video area 304, a current frequency 
(broadcast or cable), and a current channel. A Tunerlnfo 
object also contains methods as shown to receive param- 
eters, to allow an applet to get default coordinates of area 
304, to get and set the channel, and to get and set the 
frequency. Details of these methods are shown in FIGS. 
9(a)-9(f). 

[0062] As shown in FIG. 8(b), a Videolnfo object contains 
data identifying a current PIP type ("Video") and coordi- 
nates of the video area 304. A Videolnfo object also contains 
methods as shown to receive parameters and to allow an 
applet to get default coordinates of area 304. Details of these 
methods are shown in FIGS. 9(g)-9(h). 

[0063] As shown in FIG. 8(c), a PIPImplementation 
object contains methods as shown in FIG. 10(a)-10(/) and 
calls several native code routines to communicate with and 
control the video chip. PIPImplementation objects receive 
PIPlnfo objects as input. 

[0064] FIGS. 9(a)-9(fc) are flow charts of exemplary meth- 
ods in the Tunerlnfo class and the Videolnfo class. These 
methods include, without limitation: 

[0065] public void setTunerlnfo (channel, fre- 
quency), which sets information into a Tunerlnfo 
object; 

[0066] public rectangle getDe fault Coordinate^ ), 
which gets the default coordinates of a Tunerlnfo 
object; 

[0067] public int getChannel( ), which gets a current 
channel of a Tunerlnfo object; 

[0068] public void setChannel(channel)( ), which 
sets a channel in a Tunerlnfo object; 

[0069] public int getFrequencyType( ), which gets a 
current frequency type (e.g., broadcast or cable) of a 
Tunerlnfo object; 

[0070] public int setFrequencyType(frequency), 
which sets a frequency type of a Tunerlnfo object; 

[0071] public void setVideoInfo( ), which sets video 
information in a Videolnfo object; 

[0072] public rectangle getDefaultCoordioates( ); 
which obtains the default coordinates of a Videolnfo 
object 

[0073] FIGS. 10(a)-10(/) are examples of methods in the 
PIPImplementation class that are called by the virtual 
remote control applet. Each of the flowcharts of FIGS. 9 
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and 10 shows steps performed by a method in the "virtual 
remote" API of the described embodiment. It should be 
understood that this API is only one example of an API that 
could operate with the present invention and that the 
described API is not to be taken in a limiting sense. 

[0074] The methods of the PIPImplementation object 
include, without limitation: 

[0075] public void setOn(boolean switchltOn), 
which tuns the video display within the web page on 
or off; 

[0076] public boolean getOn( ), which returns an 
indicator of whether the video display within the web 
page is on; 

[0077] public void pipParms(pipparms pipparms), 
which sets up parameters in the PIPInfo objects; 

[0078] public void setCoordinates (rectangle coordi- 
nates), which sets coordinates on the video display 
within the web page; 

[0079] public Rectangle getCoordinates( ), which 
returns the coordinates of the video display wilhio 
the web page; 

[0080] public void setPIPInfo(PipInfo piplnfo), 
sends a PIPInfo class object to a PIP object. 

[0081] FIG. 11 is an exemplary list of native code func- 
tions called by the objects of FIG. 8. These functions make 
up a "native code API." The native code API will vary 
somewhat, depending on the capabilities of the video chip 
used and the capabilities required by the virtual remote 
control API. 

[0082] FIG. 12 is an example of HTML for a Web page 
that contains an example of the virtual remote control applet. 
This example is not to be taken as a limitation of the present 
invention. 

[0083] In summary, the described embodiment of the 
present invention provides a way for video data to be 
displayed on a web page under control of an applet. Because 
the applet defines the appearance and functionality of a 
virtual remote control displayed on the web page, the 
appearance and functionality of the virtual remote control 
can be easily changed without having to modify the hard- 
ware of software that is built in to the platform on which the 
applet is executed. The appearance and functionality of the 
virtual remote control are automatically updated each time a 
new version of the applet is loaded, An exemplary API for 
a set top box is described. The applet calls API functions in 
response to user input via the virtual remote control dis- 
played on the web page. 

[0084] While the invention has been described in conjunc- 
tion with a specific embodiment, it is evident that many 
alternatives, modifications and variations will be apparent to 
those skilled in the art in light of the foregoing description. 
Accordingly, it is intended to embrace all such alternatives, 
modifications and variations as fall within the spirit and 
scope of the appended claims and equivalents. 



What is claimed is: 

1. A method of controlling display of video data, com- 
prising: 

creating, by an applet, a PIP object and a PIPInfo object; 

sending the PIPInfo object to the PIP object to initially 
control display of the video data within a web page; 

receiving, by the applet, user input via a virtual control 
panel displayed by the applet on the web page; and 

calling, by the applet, virtual control API functions in 
accordance with the user input to communicate with the 
PIP object and the PIPInfo object to control the display 
of video data within the web page. 

2. The method of claim 1, wherein the display functions 
include switching between TV and video input. 

3 The method of claim 1, wherein the display functions 
include switching between broadcast frequency and cable 
frequency. 

4. The method of claim 1, wherein the display functions 
include switching between ON and OFF. 

5. The method of claim 1, wherein the display functions 
include changing the channel. 

6. An apparatus that controls display of video data, 
comprising: 

software circuitry configured to create, by an applet, a PIP 
object and a PIPInfo object; 

software circuitry configured to send the PIPInfo object to 
the PIP object to initially control display of the video 
data within a web page; 

software circuitry configured to receive, by the applet, 
user input via a virtual control panel displayed by the 
applet on the web page; and 

software circuitry configured to call, by the applet, virtual 
control API functions in accordance with the user input 
to communicate with the PIP object and the PIPInfo 
object to control the display of video data within the 
web page. 

7. A computer program product, comprising: 

a computer usable medium having computer readable 
code embodied therein for allowing an applet to control 
display of video data within a web page, including: 

computer readable program code devices configured to 
create, by an applet, a PIP object and a PIPInfo object; 

computer readable program code devices configured to 
send the PIPInfo object to the PIP object to initially 
control display of the video data within the web page; 

computer readable program code devices configured to 
receive, by the applet, user input via a virtual control 
panel displayed by the applet on the web page; and 

computer readable program code devices configured to 
call, by the applet, virtual control API functions in 
accordance with the user input to communicate with the 
PIP object and the PIPInfo object to control the display 
of video data within the web page. 

+ * + * 
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